refactor!: compose exact current-format readers - #8024
Conversation
…mat-identity # Conflicts: # rust/lance-file/src/reader.rs # rust/lance/src/dataset/transaction.rs
… xuanwo/exact-version-stack-05-file-runtime
…rsion-stack-02-fixtures
…ct-version-stack-03-v1
…sion-stack-04-encoding-mechanisms
… xuanwo/exact-version-stack-05-file-runtime
…/exact-version-stack-06-writers
…t-version-stack-07-readers
github-actions
Bot
added
A-python
…stack-07-readers # Conflicts: # rust/lance-encoding/benches/common/mod.rs # rust/lance-encoding/src/array_encoding/physical/bitpack.rs # rust/lance-encoding/src/array_encoding/physical/fixed_size_binary.rs # rust/lance-encoding/src/array_encoding/strategy.rs # rust/lance-encoding/src/compression.rs # rust/lance-encoding/src/encoder.rs # rust/lance-encoding/src/encodings/logical/primitive/sparse/writer.rs # rust/lance-encoding/src/lib.rs # rust/lance-encoding/src/testing.rs # rust/lance-file/src/lib.rs # rust/lance-file/src/reader.rs # rust/lance-file/src/version.rs # rust/lance-file/src/versions/mod.rs # rust/lance-file/src/versions/v1/encoding/plain.rs # rust/lance-file/src/versions/v1/mod.rs # rust/lance-file/src/versions/v2_0/mod.rs # rust/lance-file/src/versions/v2_0/writer.rs # rust/lance-file/src/versions/v2_1/compression.rs # rust/lance-file/src/versions/v2_1/mod.rs # rust/lance-file/src/versions/v2_2/compression.rs # rust/lance-file/src/versions/v2_2/mod.rs # rust/lance-file/src/versions/v2_3/compression.rs # rust/lance-file/src/versions/v2_3/mod.rs # rust/lance-file/src/writer.rs # rust/lance-file/src/writer/structural.rs # rust/lance-file/src/writer_tests.rs # rust/lance-file/test_data/exact_versions/README.md # rust/lance-index/src/scalar/lance_format.rs # rust/lance-index/src/vector/utils.rs # rust/lance-table/src/format.rs # rust/lance-table/src/format/manifest.rs # rust/lance/src/dataset.rs # rust/lance/src/dataset/builder.rs # rust/lance/src/dataset/fragment.rs # rust/lance/src/index/vector/ivf.rs # rust/lance/src/index/vector/ivf/io.rs # rust/lance/src/index/vector/pq.rs
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
There was a problem hiding this comment.
The exact-version reader roots address a real file-format boundary: V2.0–V2.3 now validate their own footer identity, encoding grammar, alignment, projection shape, and metadata capabilities while shared I/O and decode machinery stays centralized. The stable wire contracts remain covered through exact-version fixtures, and I found no durable-format or correctness blocker.
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Xuanwo
changed the title
| if gbo_table.is_empty() { | ||
| return Err(Error::internal( | ||
| "File did not contain any global buffers, schema expected".to_string(), | ||
| )); | ||
| } | ||
| let schema_start = gbo_table[0].position; | ||
| let schema_size = gbo_table[0].size; | ||
|
|
||
| let num_footer_bytes = file_len - schema_start; |
There was a problem hiding this comment.
Could we validate the GBO descriptors before using them to build ranges? Here schema_start comes from the file and is used in file_len - schema_start before finish_metadata applies the V2.1+ alignment check. A malformed file can therefore underflow here (or wrap in release) and turn into an oversized read/allocation instead of InvalidInputSource. The indexed-metadata path has the same issue with position + size. Please validate alignment and file bounds with checked arithmetic immediately after decoding the GBO table, before either path performs I/O.
There was a problem hiding this comment.
Should this be a follow-up issue? This series is a one-to-one mapping of the old logic and was not intended to change anything else except the code layout.
## Why Main CI broke after #8024 landed: `ReaderProjection::from_column_names` / `from_whole_schema` were removed, but IVF index call sites still used them, so Rust, Java JNI, and Python builds failed to compile. Separately, newly published RUSTSEC advisories caused `cargo-deny` to fail on unpinned transitive crates. ## What - Switch remaining IVF readers/tests to `lance_file::versions::reader_projection_from_*` - Bump fixable transitive deps (`anyhow`, `event-listener`, `memmap2`, `rkyv`, `spin`) in all three lockfiles - Ignore `RUSTSEC-2026-0002` for `lru` 0.12.x until `goosefs-sdk` upgrades
2 participants
Part 7/12 of #7877. Depends on #8023.
This is an independently reviewable step toward the final layout demonstrated in #7979.
This PR gives V2.0, V2.1, V2.2, and V2.3 explicit reader composition alongside V1. The root reader keeps shared runtime machinery and type-erased forwarding, but exact version modules own construction, decoder selection, and supported capabilities.
The large shared reader implementation is reduced to mechanisms that are actually common. Version-sensitive choices no longer depend on ordering comparisons or scattered conditions, and the reader still parses persisted identity once at the boundary.
Validation:
cargo test -p lance-filecargo clippy --all --tests --benches -- -D warningsuv run make buildinpython/cargo check --manifest-path java/lance-jni/Cargo.toml